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ABSTRACT 

A rationale and illustration of the use of original 
records or protocol materials in an evaluation research ?:eport are 
described. Records of school observations and audiotape tra:iscripts 
were selected to represent the concepts or categories which were 
developed in the process of evaluation* These qualitative data were 
collected in a project which used a variety of methods, anc*, in 
conjunction with quantitative data, formed the basis of an evaluation 
report somewhat analogous to a case study. This view of evaluative 
research within the case study tradition should, in the Ion.:; run, 
provide a more detailed analysis and understanding of the pi.ocess and 
outcomes of education* (Author/BB) 
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This paper will describe the rationale and use of illustrative pro- 
tocol material in an evaluation report. The terra protocol is used here 
in the sense of an original record--a transcript, detailed observational 
notes, audio or video tapes, from vmich later analyses and inferences are 
drawn. The terx protocol also coroiotcs selectivit;y in the sense that 
B, Othanel Smith (1969) conveyed in his discussion of the use of protocol 
materials in teacher preparation. Recordings of behavioral situations 
v/ere to be selected to represent specific dimensions of settings in 
which behavior occurs and to represent the category or concept the 
recordings are intended to illustrate. Siuilarly , the records of school 
observations and audiotape transcripts represent the concepts or cate- 
gories which evolved from or v/ere used in the evaluation research study. 
The rationale for the use of oualito,tive data in evaluative research, 
and for presenting protocol material or original records in the evalu- 
ation report is discussed belov;. 

Rationale 

Recent work in evaluation has held the premise that knowledge in 
• education, and more particularly evaluation theor:::^^ •'^^^d. methodology, can 
be advanced by the application of the v:ays of knowing--the styles of 
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conceptualization and techniques of acquiring infor:aatiou--of several 
disciplines. This trend would bring evaluative research in education 
more firmly within the traditional patterns of inquiry within the social 
sciences. 

Diesin^ (1971) has characterized and discussed these various methods 
v/ithin tv;o broad categories: formal methods and theories, and partici- 
pant-observer and clinical methods. He has called these methods "pat- 
terns of discovery," since he is dealing with the v.'hole process of 
inquiry, including the creatiog and developing of knowledge, and not 
just the verification of knowledge. 3rv<v\*^i (-I966) has analyzed the prob- 
lem of inouir^/ in a different manner, but also contrasts tv/o major 
approaches within sociological research, one representing the quanti- 
tative and the other the qualitative study of hum.n phenomena. V/ithin 
the quantitative research tradition, there are a variety of approaches, 
including traditional empiricism. In qualitative research, there are 
more interpretive types of inquiry, including pa,rticipant observation. 
Glascr nnd otrr.uss (1^6?) have attempted to document and explain the 
usefulness of qualitative research for "The Discovery of Grounded Theory. " 

In educational evaluation, Louis Smith r.nd his colleagues have begun 
to demonstrate the usefulness o7 techni^ue3 from p::ycliology, sociology 
-nd anthropologry. omith and .'.'hlr:nd (117^-) have described the con- 
tributions of miethcds from these disciplines in the evaluation of a CAT 
" project. Berlak (1971), Solono.-: ■1971), ^-nd .''^eif (l97l) used partici- 
pant observation in curriculum research r lA evalua'.ion projects. The 
1973 r:enucst for Evaluation Proioosals in the IJxperimental Schools 

Program included provisions for the development of narrative case studies 
of these projects. 
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As discussed in another presentation (Tittle, 1973), the use cT 
variety of methods and concepts from other disciplines is undertc^.en :.n 
educational evaluation to go beyond the nininiu.Ti paradic-i for cv':luntion 
of definir^ objectives and r.;easuring pupil change before and ^i: or the 
provision of learning experiences. The alternative viev; reouircs r- 
change, or at least an addition to, the current oaradi^pi for evaluation 
based on experimental design. The change in perspective for evaluation 
is intended to provide a better understandir\3 of the context for evalu- 
atidn, and of the process which intervenes betv/een the first and second 
assessment of behavior. The need to encompass a variety of methods lead 
to an analogy with a case svudy, where careful, extc:;sive descriptions 
are developed. This alternative paradij::; for evaluative research moves 
tovrard the more detailed analysis and undcrstandi.-;':; of settings in educa 
tion. The application of concepts and methods from v. variety of dis- 
ciplines in the case study tradition seems to be an rilternative which 
will eventually permit a better underst-:.ndii\':: of the variables involved 
in effecting pupil change or chan'-e in the beho.vior of other persons 
involved in education. rVentually , these series of case studies might 
permit "cross setting" evaluative research, from the descriptions and 
coding categories v;hich v;oula be developed, much as cross cultural 
research is conducted from the anthropological area files. 

Use of Protocol v:aterial 

As jr. Spilman^s paper (197^0 indicates, it is possible to work v;it 
Qualitative data, beginniiig in an intuitive but reflective manner, and 
evolve categories for which there is consensus in co-ling. In one sense 
this is nothing more than the usual application of content analysis to a 
series of semina.r transcripts. Hov;ever, the application in evaluation 



differs in a voi^ fundamental senze: tha\, is the sense in which the 
evaluator must arrive at an cnssessment of the r^.eriz or worth of various 
effects and outcomes. In this instance, the assessment v/as of the out- 
comes of a seminar for the faculty <?ttendinc and for the documents v.^iich 
the faculty produced. In this eva].uation project, the evaluator had to 
be both inquiry oriented, to develop knov;led5j:e and understanding, of what 
happened in terms of emotional irvberchanrjes - nd tVio resulting q_ualitative 
records, and at the sam.e time realize th.^: ': the results of the ino^uiry 
must permit evaluation-- judgments of value or v/orth. The catecory system 
(reported in rjpilman, 197^0 deve.loped from this particular type of in- 
quiry, and the categories were foreordained to be evaluative: Mow had 
the purposes of the seminar been explained or conceived, both by Dr. Marcos 
and the fa^c^ulty involv-jd? vrnat w-tr: ,^ood or bad about this seminar, i.e., 
what intuitively came throu^n as good or bad e:q)eric:ices of the seminar? 
A more general statement of these categories would toke the form, Ivhat 
v%ar- good (or end)? Vlhy v;as it good (or bad)'? Miat was good (or bad) in 
relation to goals or objectives': 

For the semdnar evaluation, it was clear that quantitative records, 
such as drop outs, absence, silence if attending, r.nd positive or negative 
reports produced by individual members, conveyed the information that the 
e>rperience was negative for mainly faculty members. Put to gain an under- 
standing of why this was the outcomie, tViree approaches were to.ken. One 
was the applicc'ition of group process analysis to a sample of audiotapes, 
the second was the interpretive ncarrative, and the third v/as the inclusion 
of protocol material in the descriptive and evaluative sections of the 
report. 



The group process analysis used the Keciprocal Category System of 
^Ober (Tarrier, 1973)- This data indicated that group maintenance behav- 
iors, such as accepting, ampliTying contributions oi another , and vr^rnis 
at:';iosphere , v;ere among the lov/est frequency of categories of leader 
behavior. Task r.ategori^: such as elicits , responds , initiates , and 
directs , accounted for about 30vj of Xarcos* remarks. From the standpoint 
of group theory, the heavy emphasis on task behavior v/as inappropriate, 
in a situation where the leader, :>:arcos, was not perceived as having all 
the knovrledge to give, and group members felt that they had something to 
bring to the seminar. 

The second approach went beyond the level of statements just made, 
to present an "interpretive narrative^' r^s the main evaluation stateir^ent 
for the !-:arcos seminar (Howell, 1973)- The interpretive narrative des- 
cribed expectations and assumptions of bcoh ;^r. Marcos and the faculty 
members, and went on to describe what acturilly happened in the seminar 
diiring the Fall, Winter, and Spring series of sessions. This inter- 
pretive narrative was dependent on the evr^luative category system men- 
tioned above. The outline for the narrative v/as erisontially the category 
system and the interpretation came from the conceptualization in the 
evaluation category system. 

The third approach v;as the inclusion of protocol material (ori- 
ginal records) in the description and evaluation sections of the report. 
Transcripts of Dr. Marcos interactions iii the faculty seminar setting, 
a.nd in various school settings, were included to illustrate the type 
and quality of interchange described in the interpretive narrative, 
group process analysis, and in school trr^ining sessions. 

This integration of protocol material serves several purposes. The 
use of original records makes concrete trie content analysis of the quali- 



tative material and also provides a samplinc of the b'..Jc data. Including 
a sampling of "basic data permits; verification or alte^^native interpreta- 
tions by other evaluators. An exm.aple of an alternative intei^retation 
of basic data is found in Li^hthall^s review (1973) and explanation of 
Snith and /Keith's v/ork (197^) docu' .cnting an orcanizational analysis of 
an innovative elementary school. 

Spence has provided a discuf^sion of the problem of analyzing and 
suTu'Tiarizing behavior in his arti.'lc on analog and digital descriptions of 
behavior. Behavior is continuous, and "categorization begins when v/e 
attach words to the pheno.nenon r :\X continues as we group the words into 
oor.ie theoretical hierarchy or schema of classification. There is no 
limit to the kinds of categories v.-e can invent, and they all have one 
thing in comnion: They can only approximate the reality we are trying to 
describe (1973, P. ^79)'" ^""'^ goes on to note that there are obvious 
limitations to an attempt to create a p-^r-T^-anent library of real events, 
since it is simply a copy of thj real world and thus no saving. If less 
than the real events are stored, then we have all the disadvantages of an 
incomplete library. VJhere data are transformed, there are errors intro- 
duced. 

In dealing v/ith the complexities of educational phenomena, the most 
useful strateg;;)'' is a compromise: the development of categories and quan- 
titative data, but retentioxn of original rcc* r^ds in sufficient detail and 
representativeness to permit their use in illustrating our samma'L*;^^ of the 
events and to permit reinterpretation. 

An Illustration 

If the interpretive narrative and group process analysis agree that 
Dr. Marcos had a particular viev/ of himself as leader, one in wliich he 
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eyi^Gcted others to find and a';;ree with his point of view, protocol mate- 
rials can nake concrete the r|^uality of this view of len.dership. Examples 
of this t^-pe of interaction were recorded in the training sessions v;ith 
teaxhers r.nd in the seminar. An exa-niple from the s^jjiLmer teacher training 
session is as follov/s: 

"...I didn^t put into circulation anything to read until 
late last v/eek. I didn't show you rr^ v/ork on reading 
and ma.thematics as such until last week (fourth v;eek 
of the five-week session), because you have nut come 
here to Icar^i m;^^ pas'o. I have not reali::ed m^-self in 
that school yet. '■rtiat I realized was that certain 
things connected wiuh a certain vision. If it is a 
tool that you can put in your hands to sa.ve time, "by all 
means; if it is not, then it doesn^t appear. I have 
done many, many thii\';s in my p<'ist I couldn't even put 
in yo^JJT hands. Now, us a servant of yours, I am 
trying to demonstrate what is the role of a leader to- 
day. It's the one who knov;s--who knov;s what there is 
to do ct every minute, which yovi can't do--you can't 
have in yo^or memory, you can't have in your past. To 
know v/hat to do nov; requires that I get now the impact, 
not che one that I received from somewhere else at 
another time. It has to be now. .. (quoted in Tittle, 
howell, and Spilman, 1973, p. 67)." 
In a meeting with the school administrrtors. Dr. Marcos presented a 
written statement evaluating the school's progress after three months of 
operation. An administrator mentioned obtaining supplies for teachers 
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ox laboratories (subjccu3 such as science^ art, etc.); Marcos replied: 
"This is the t/jc ol' thiiig I a:n tallvin:;; about. If the 
teachers t,:oy they v;:-::t l-elp, then we arc not inter- 
Terir.:;, but are supporting the;.i. This is v/hy I take the 
initiative in writir:;^ view of the school, and perhaps 
everyone could, because it is complenientcry . All know, 
and if ever;^^one trios to talce tlieir bearing after these 
three rr.onths, it will be helpful. I speal^: as someone who 
sees the forest. .. (r;uoted in Tittle, Kowell, and Spilman, 
1973, p. 76)." 

Dv<riii^ xhe early part of the project, detailed suraraaries of the faculty 
seminar a.eetin{;s were prepared. The sa-mary for October 20, 1971, shows 
how individual quotations v/ere integrated v/ith the sunrmiy: 

".../m additional question that came to the fore from 
time to time was that of Dr. T-arcos' role v/ith respect 
to the seminar. At one point, early in the period, he 
said, '^fnat I feel at this time is one voice, like yoirrs, 
and I'll express myself ^ust as one person in this seminar." 
That, however, vms evidently not the perception, nor was 
it the desire of the ,^roup. I^ater, the fourth faculty 
member (above) ur^ed Dr. Marcos to "take the leadership 
of the seminar," but he declined, sayin^j that he v/as 
present as a guest, although he had earlier admitted 
having a part in "steering" the group. Toward the end of 
the discussion another professor asked hi::\, "Are you 
chairing this seminar or not?" lie made it plain that 
he was not, but shortly afterv/ard (the problem of chair- 
manship having been left unresolved) he malce a transition 
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to an instructional T.ode of operation in v;aich he becan 
to vlescribe hiG :;icthod of r:ieasurin^'^ the ''cost of educa- 
tion" to the pupils (r:uoted in Tittle, llowell and 
oplltnan, 1973, PP. 96-97)." 

This P'V-.per has presented rationale and illustration of the use of 
criminal records or protocol r::aterials in an evaluation research report. 
i\ecords of school observations and audiotape transcripts were selected to 
rcpreserrc the concepts or categories which were developed in the process 
of evaluation. These qualitative data were collected in a project v/hich 
used a variety of raethods, and, in conjunction with 'i^jaantitative data, 
for.r.ed the basis of an evaluation report somewhat analogous to a case 
study. This viev/ of evaluative research v/ithin the case study tradition 
should, in the lon^ run, provide a mora detailed analysis and under- 
standing of the process and outcoT.es of education. 
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